Driving assistance device, driving assistance method, and storage medium

ABSTRACT

A driving assistance device of an embodiment includes a recognizer that recognizes a surroundings situation of a vehicle, and a display controller that causes an image for assisting an occupant driving the vehicle to be displayed in a plurality of preset display modes on a display device, the plurality of display modes include display modes corresponding to at least a display mode corresponding to a mode in which the vehicle travels on a narrow road, and the display controller causes transition to any one of the plurality of display modes to be performed on the basis of the surroundings situation recognized by the recognizer.

CROSS-REFERENCE TO RELATED APPLICATION

Priority is claimed on Japanese Patent Application No. 2022-058112,filed on Mar. 31, 2022, the content of which is incorporated herein byreference.

BACKGROUND Field of the Invention

The present invention relates to a driving assistance device, a drivingassistance method, and a storage medium.

Description of Related Art

In recent years, efforts to provide access to a sustainabletransportation system in consideration of vulnerable people amongtraffic participants have become active. In order to realize this,research and development for further improving the safety andconvenience of transportation through research and development regardinga driving assistance technology has been focused upon. In relationthereto, a technology for determining whether or not assistance forpassing-driving of a host vehicle is to be permitted on the basis of aspeed of an oncoming vehicle when a host vehicle and an oncoming vehiclewill pass each other in a passing state on a narrow road, and causing animage captured by a side camera to be displayed on a display when adetermination is made that assistance for passing-driving is to bepermitted is known (for example, Japanese Unexamined Patent Application,First Publication No. 2020-149204).

SUMMARY

Incidentally, in a driving assistance technology, appropriate imagesaccording to various other surroundings situations cannot be displayedin some cases.

Aspects of the present invention have been made in consideration of suchcircumstances, and one object thereof is to provide a driving assistancedevice, a driving assistance method, and a storage medium capable ofperforming more appropriate driving assistance according to asurroundings situation. Further, this will contribute to the developmentof a sustainable transportation system.

The driving assistance device, driving assistance method, and storagemedium according to the present invention adopt the followingconfigurations.

-   -   (1): A driving assistance device according to an aspect of the        present invention includes a recognizer configured to recognize        a surroundings situation of a vehicle; and a display controller        configured to cause an image for assisting an occupant driving        the vehicle to be displayed in a plurality of preset display        modes on a display device, wherein the plurality of display        modes include display modes corresponding to at least a first        mode in which the vehicle travels in a direction in which the        road extends, a second mode in which the vehicle travels at an        intersection, a third mode in which the vehicle travels in a        merging section, a fourth mode in which the vehicle travels on a        narrow road, and a fifth mode in which the vehicle travels while        avoiding contact with objects, and the display controller causes        transition to any one of the plurality of display modes to be        performed on the basis of the surroundings situation recognized        by the recognizer.    -   (2): In the aspect (1), the display controller selects an image        to be displayed, from among at least an image imitating the        vehicle, an image showing a position of an object around the        vehicle, an image showing a direction in which the vehicle        should travel, a target position serving as a lane change        destination of the vehicle, an image for prompting acceleration        or deceleration, and an image for prompting the occupant to        perform a steering operation on the basis of a display mode        transitioning on the basis of the surroundings situation        recognized by the recognizer, and causes the image to be        displayed.    -   (3): In the aspect (1), the display controller causes an image        imitating the vehicle in the third mode and the fourth mode to        be displayed as an image when the vehicle is viewed from above,        and causes an image imitating the vehicle in the first mode, the        second mode, and the fifth mode to be displayed as an image when        the vehicle is viewed from behind.    -   (4): In the aspect (1), the display controller causes an image        imitating the vehicle to be displayed when the display        controller causes an image to be displayed in a display mode        corresponding to the first mode, and causes an image showing a        direction to be watched by the occupant to be displayed when an        object is present around the vehicle and the object is not        present in a line-of-sight direction of the occupant, and causes        an image showing a direction of the object viewed from the        vehicle to be displayed when the object is present in the        line-of-sight direction of the occupant.    -   (5): In the aspect (1), the display controller causes an image        imitating the vehicle to be displayed when the display        controller causes an image to be displayed in a display mode        corresponding to the second mode, causes an image showing a        relative position of an object with respect to the vehicle to be        displayed when there is the object likely to come into contact        with the vehicle at the intersection, and causes an image        showing a direction to be watched by the occupant to be        displayed when the object is located in a traveling direction of        the vehicle M.    -   (6): In the aspect (1), the display controller causes an image        imitating the vehicle to be displayed when the display        controller causes an image to be displayed in a display mode        corresponding to the third mode, causes another vehicle        traveling in a lane serving as a lane change destination to be        displayed as an image with a first predetermined color and        causes an image showing a target position of the lane change        destination to be displayed as an image with a second        predetermined color, and causes the image with the first        predetermined color and the image with the second predetermined        color to be displayed on the side in which the vehicle performs        lane change with respect to the image imitating the vehicle    -   (7): In the aspect (6), the display controller causes an image        for accelerating or decelerating the vehicle on the basis of a        positional relationship between the vehicle and the target        position to be displayed when the display controller causes the        image to be displayed in the display mode corresponding to the        third mode.    -   (8): In the above aspect (1), the display controller causes an        image imitating the vehicle to be displayed when the display        controller causes an image to be displayed in a display mode        corresponding to the fourth mode, and causes a direction in        which the vehicle should travel to be displayed and causes an        image for prompting the occupant to perform a steering operation        to be displayed when there is an object around the vehicle.    -   (9): In the above aspect (1), the display controller causes an        image imitating the vehicle to be displayed when the display        controller causes an image to be displayed in a display mode        corresponding to the fifth mode, and causes an image showing a        direction in which the vehicle should travel with respect to the        image imitating the vehicle to be displayed, and causes an area        other than the direction in which the vehicle should travel        around the image imitating the vehicle to be displayed in a        background color different from those displayed in the first to        fourth modes.    -   (10): A driving assistance method according to an aspect of the        present invention includes recognizing, by a computer, a        surroundings situation of a vehicle; causing, by the computer,        an image for assisting an occupant driving the vehicle to be        displayed in a plurality of preset display modes on a display        device, the plurality of display modes including display modes        corresponding to at least a first mode in which the vehicle        travels in a direction in which the road extends, a second mode        in which the vehicle travels at an intersection, a third mode in        which the vehicle travels in a merging section, a fourth mode in        which the vehicle travels on a narrow road, and a fifth mode in        which the vehicle travels while avoiding contact with objects;        and causing, by the computer, transition to any one of the        plurality of display modes to be performed on the basis of the        recognized surroundings situation.    -   (11): A storage medium according to an aspect of the present        invention is a computer-readable non-transitory storage medium        having a program stored therein, the program causing a computer        to: recognize a surroundings situation of a vehicle; cause an        image for assisting an occupant driving the vehicle to be        displayed in a plurality of preset display modes on a display        device, the plurality of display modes including display modes        corresponding to at least a first mode in which the vehicle        travels in a direction in which the road extends, a second mode        in which the vehicle travels at an intersection, a third mode in        which the vehicle travels in a merging section, a fourth mode in        which the vehicle travels on a narrow road, and a fifth mode in        which the vehicle travels while avoiding contact with objects;        and cause transition to any one of the plurality of display        modes to be performed on the basis of the recognized        surroundings situation.

According to the aspects (1) to (11), it is possible to perform moreappropriate driving assistance according to the surroundings situation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a configuration diagram of a vehicle system using a drivingassistance device according to an embodiment.

FIG. 2 is a diagram for describing a function of a determiner.

FIG. 3 is a diagram for describing a first display mode in a normalmode.

FIG. 4 is a diagram for describing a second display mode in the normalmode.

FIG. 5 is a diagram for describing a third display mode in the normalmode.

FIG. 6 is a diagram for describing a fourth display mode in the normalmode.

FIG. 7 is a diagram for describing a fifth display mode in the normalmode.

FIG. 8 is a diagram for describing a sixth display mode in the normalmode.

FIG. 9 is a diagram for describing a first display mode in anintersection mode.

FIG. 10 is a diagram for describing a second display mode in theintersection mode.

FIG. 11 is a diagram for describing a first display mode in a mergingmode.

FIG. 12 is a diagram for describing a second display mode in the mergingmode.

FIG. 13 is a diagram for describing a third display mode in the mergingmode.

FIG. 14 is a diagram for describing a fourth display mode in the mergingmode.

FIG. 15 is a diagram for describing a first display mode in a narrowroad passage mode.

FIG. 16 is a diagram for describing a second display mode in the narrowroad passage mode.

FIG. 17 is a diagram for describing a third display mode in the narrowroad passage mode.

FIG. 18 is a diagram for describing a first display mode in an emergencyavoidance mode.

FIG. 19 is a flowchart showing an example of a flow of processing thatis executed by the driving assistance device of the embodiment.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of a driving assistance device, a drivingassistance method, and a storage medium of the present invention will bedescribed with reference to the drawings. Hereinafter, a case in which aleft-hand traffic regulation is applied will be described, but when aright-hand traffic regulation is applied, the right and the left may bereversed.

[Overall Configuration]

FIG. 1 is a configuration diagram of a vehicle system 1 using a drivingassistance device according to an embodiment. A vehicle on which thevehicle system 1 is mounted (hereinafter referred to as a host vehicleM) is, for example, a vehicle such as a two-wheeled vehicle, athree-wheeled vehicle, or a four-wheeled vehicle, and a driving sourcethereof includes an internal combustion engine such as a diesel engineor a gasoline engine, an electric motor, or a combination thereof. Theelectric motor operates using power generated by a power generatorconnected to the internal combustion engine or discharge power of asecondary battery or a fuel cell. Hereinafter, the driving assistancedevice may be applied to an automated driving vehicle. The automateddriving is, for example, to automatically control one or both ofsteering and acceleration or deceleration of the host vehicle M toexecute driving control. Examples of the driving control of the hostvehicle M may include various driving assistance such as adaptive cruisecontrol (ACC), auto lane changing (ALC), lane keeping assistance system(LKAS), forward collision warning (FCW), and collision mitigationbraking system (CMBS). The automated driving vehicle may be a vehiclewhose driving is partially or wholly controlled by manual driving of anoccupant (driver).

The vehicle system 1 includes, for example, a camera (an example of animager) 10, a radar device 12, a light detection and ranging (LIDAR) 14,an object recognition device 16, a communication device 20, a humanmachine interface (HMI) 30, a vehicle sensor 40, a navigation device 50,a driver monitor camera 70, a driving operator 80, a driving assistancedevice 100, a travel driving force output device 200, a brake device210, and a steering device 220. These devices or equipment are connectedto each other by a multiplex communication line such as a controllerarea network (CAN) communication line, a serial communication line, awireless communication network, or the like. The configuration shown inFIG. 1 is merely an example, and a part of the configuration may beomitted or other configurations may be added thereto. A combination ofthe camera 10, the radar device 12, and the LIDAR 14 is an example of an“external sensor ES”. The external sensor ES may include anotherdetector (for example, sonar) that recognizes a surroundings situationof the vehicle, and may include the object recognition device 16. Theexternal sensor ES may be configured of only the camera 10 or may have asimple configuration with only the camera 10 and the radar device 12.The HMI 30 is an example of an “output device”.

The camera 10 is, for example, a digital camera using a solid-stateimaging device such as a charge coupled device (CCD) or a complementarymetal oxide semiconductor (CMOS). The camera 10 is attached to anylocation on a host vehicle M. When a forward side of the host vehicle Mis imaged, the camera 10 is attached to, for example, an upper portionof a front windshield, a rear surface of a rearview mirror, or the like.When a backward side of the host vehicle M is imaged, the camera 10 isattached to an upper portion of a rear windshield, a back door, or thelike. When a sideward side and a rear sideward side of the host vehicleM are imaged, the camera 10 is attached to a door mirror or the like.The camera 10, for example, periodically and repeatedly imagessurroundings of the host vehicle M. The camera 10 may be a stereocamera.

The radar device 12 radiates radio waves such as millimeter waves to thesurroundings of the host vehicle M and detects radio waves (reflectedwaves) reflected by an object to detect at least a position (a distanceand orientation) of the object. The radar device 12 is attached to anylocation on the host vehicle M. The radar device 12 may detect aposition and a speed of the object using a frequency modulatedcontinuous wave (FM-CW) scheme.

The LIDAR 14 irradiates the surroundings of the host vehicle M withlight (or an electromagnetic wave having a wavelength close to that oflight) and measures scattered light. The LIDAR 14 detects a distance toa target on the basis of a time from light emission to light reception.The light to be radiated is, for example, pulsed laser light. The LIDAR14 is attached to any location on the host vehicle M.

The object recognition device 16 performs sensor fusion processing ondetection results of some or all of the camera 10, the radar device 12,and the LIDAR 14 included in the external sensor ES to recognize aposition, type, speed, and the like of the object. The objectrecognition device 16 outputs recognition results to the drivingassistance device 100. The object recognition device 16 may outputdetection results of the camera 10, the radar device 12, and the LIDAR14 as they are to the driving assistance device 100. The objectrecognition device 16 may be omitted from the vehicle system 1.

The communication device 20, for example, communicates with anothervehicle present around the host vehicle M using a cellular network, aWi-Fi network, Bluetooth (registered trademark), dedicated short rangecommunication (DSRC), or the like or communicates with various serverdevices via a wireless base station.

The HMI 30 presents various types of information to the occupant of thehost vehicle M under the control of an HMI controller 140, and receivesinput operations by the occupant. The HMI 30 includes, for example, adisplay device 32. The display device 32 is, for example, a liquidcrystal display (LCD) or an organic electro luminescence (EL) displaydevice. The display device 32 is provided, for example, near the frontof a seat of a driver (a seat closest to a steering wheel) in aninstrument panel, and is installed at a position visible to the occupantthrough a gap in the steering wheel or through the steering wheel. Thedisplay device 32 may be installed at a center of the instrument panel.The display device 32 may be a head up display (HUD). The HUD projectsan image onto a portion of the front windshield in front of the seat ofthe driver, thereby allowing eyes of the occupant seated on the seat ofthe driver to visually recognize a virtual image. The display device 32displays images that are generated by the HMI controller 140, which willbe described below. The HMI 30 may include, for example, a speaker, aswitch, a microphone, a buzzer, a touch panel, a key. The HMI 30 mayinclude, for example, a driving changeover switch for switching betweenautomatic driving and manual driving by the occupant. The switchincludes, for example, a blinker switch (a direction indicator). Theblinker switch is provided, for example, on a steering column or asteering wheel. The blinker switch is an example of an operator thatreceives an instruction to change the lane of the host vehicle M fromthe occupant, for example. The switch may include a switch for adjustinga set speed of the host vehicle M.

The vehicle sensor 40 includes, for example, a vehicle speed sensor thatdetects a speed of the host vehicle M, an acceleration sensor thatdetects an acceleration, a yaw rate sensor that detects an angular speedaround a vertical axis, and an orientation sensor that detects adirection of the host vehicle M. The vehicle sensor 40 may include asteering angle sensor that detects a steering angle of the host vehicleM (which may be an angle of a steered wheel or an operating angle of thesteering wheel). The vehicle sensor 40 may include a position sensorthat acquires a position of the host vehicle M. The position sensor is,for example, a sensor that acquires position information (longitude andlatitude information) from a global positioning system (GPS) device. Theposition sensor may be a sensor that acquires position information usinga global navigation satellite system (GNSS) receiver 51 of thenavigation device 50.

The navigation device 50 includes, for example, a global navigationsatellite system (GNSS) receiver 51, a navigation HMI 52, and a routedeterminer 53. The navigation device 50 holds map information 54 in astorage device such as a hard disk drive (HDD) or a flash memory. TheGNSS receiver 51 specifies a position of the host vehicle M on the basisof a signal received from a GNSS satellite. The position of the hostvehicle M may be specified or complemented by an inertial navigationsystem (INS) using an output of the vehicle sensor 40. The navigationHMI 52 includes a display device, a speaker, a touch panel, keys, andthe like. The navigation HMI 52 may be partly or wholly shared with theHMI 30 described above. The route determiner 53, for example, determinesa route (hereinafter, an on-map route) from the position of the hostvehicle M specified by the GNSS receiver 51 (or any input position) to adestination input by the occupant using the navigation HMI 52 byreferring to the first map information 54. The map information 54 is,for example, information in which a road shape is represented by linksindicating roads and nodes connected by the links. The map information54 may include a curvature of the road, point of interest (POI)information, and the like. The map information 54 may include, forexample, lane center information or lane boundary information, and mayinclude road information, traffic regulation information, addressinformation (an address and postal code), facility information,telephone number information, and the like. The map information 54 maybe updated at any time by the communication device 20 communicating withanother device.

The navigation device 50 may perform route guidance using the navigationHMI 52 on the basis of the on-map route. The navigation device 50 may berealized, for example, by a function of a terminal device such as asmartphone or a tablet terminal possessed by the occupant. Thenavigation device 50 may transmit a current position and a destinationto a navigation server via the communication device 20 and acquire thesame route as the on-map route from the navigation server.

The driver monitor camera 70 is, for example, a digital camera using asolid-state imaging device such as a CCD or CMOS. The driver monitorcamera 70 is attached to any location on the host vehicle M at aposition and orientation at which a head of an occupant (hereinafterreferred to as a driver) seated in a seat of the driver of the hostvehicle M can be imaged from the front (in an orientation in which aface is imaged). For example, the driver monitor camera 70 is attachedto an upper part of the display device provided in the central part ofthe instrument panel of the host vehicle M. The driver monitor camera 70outputs, to the driving assistance device 100, an image obtained byimaging the vehicle cabin including the driver of the host vehicle Mfrom a position at which the driver monitor camera 70 is disposed.

The driving operator 80 includes, for example, an accelerator pedal, abrake pedal, a shift lever, and other operators, in addition a steeringwheel. A sensor that detects an amount of operation or the presence orabsence of an operation is attached to the driving operator 80, and adetection result thereof is applied to the driving assistance device100, or some or all of the travel driving force output device 200, thebrake device 210, and the steering device 220. The steering wheel is anexample of “an operator that receives a steering operation by a driver”.The operator does not necessarily have to be annular, and may be in theform of a modified steering wheel, joystick, buttons, or the like. Asteering grip sensor 84 is attached to the steering wheel 82. Thesteering grip sensor 84 is realized by a capacitive sensor or the like,and outputs, to the driving assistance device 100, a signal capable ofdetecting whether or not the driver is gripping the steering wheel 82(which means that the driver is in contact with the steering wheel 82 ina state in which force can be applied). The steering wheel may beprovided with a mechanism that applies a reaction force for steering ina predetermined direction (or not for steering in the predetermineddirection) to the occupant (driver) under the control of the drivingassistance device 100.

The driving assistance device 100 includes, for example, a recognizer110, a determiner 120, a driving controller 130, an HMI controller 140,and a storage 150. The recognizer 110, the determiner 120, the drivingcontroller 130, and the HMI controller 140 are realized, for example, bya hardware processor such as a central processing unit (CPU) executing aprogram (software). Some or all of these components may be realized byhardware (circuit; including circuitry) such as a large scaleintegration (LSI), an application specific integrated circuit (ASIC), afield-programmable gate array (FPGA), or a graphics processing unit(GPU), or may be realized by software and hardware in cooperation. Theprogram may be stored in a storage device (a storage device including anon-transitory storage medium) such as an HDD or a flash memory of thedriving assistance device 100 in advance or may be stored in adetachable storage medium such as a DVD or a CD-ROM and installed in theHDD or flash memory of the driving assistance device 100 by the storagemedium (a non-transitory storage medium) being mounted in a drivedevice. The HMI controller 140 is an example of a “display controller”.

The storage 150 may also be realized by the various storage devices, asolid state drive (SSD), an electrically erasable programmable read onlymemory (EEPROM), a read only memory (ROM), a random access memory (RAM),or the like. The storage 150 stores, for example, programs and othervarious types of information. The map information 54, for example, maybe stored in the storage 150.

The recognizer 110 recognizes a state such as a position, speed, oracceleration of an object present around the host vehicle M (within apredetermined distance from the host vehicle M) on the basis ofinformation input from the external sensor ES. The object is, forexample, a traffic participant such as another vehicle, a bicycle, and apedestrian. The position of the object, for example, is recognized as aposition at absolute coordinates with a representative point (acentroid, a drive shaft center, or the like) of the host vehicle M as anorigin, and is used for control. The position of the object may berepresented by a representative point such as a centroid or a corner ofthe object or may be represented by an area. The “state” of the objectmay include an acceleration or jerk of the object, or an “action status”(for example, whether or not the object is changing lanes or is about tochange lanes). The recognizer 110 may recognize a type (another vehicle,a bicycle, or a pedestrian) of the object, or the like on the basis offeature information such as a size, shape, and color of the object.

The recognizer 110 recognizes, for example, a lane in which the hostvehicle M is traveling (traveling lane). For example, the recognizer 110recognizes left and right demarcation lines of the host vehicle M from acamera image captured by the camera 10, and recognizes the travelinglane on the basis of positions of the recognized demarcation lines. Therecognizer 110 may recognize targets (a traveling road boundary or aroad boundary) that can specify a lane position including a roadshoulder, a curb, a median strip, a guard rail, a fence, a wall, or thelike as well as the demarcation lines, to recognize the traveling lane.In this recognition, the position of the host vehicle M acquired fromthe navigation device 50 or a processing result of the INS may beadditionally considered. The recognizer 110 recognizes a temporary stopline, an obstacle, a red signal, a toll gate, and other road events.

The recognizer 110 recognizes a position or posture of the host vehicleM with respect to the traveling lane when recognizing the travelinglane. The recognizer 110 may recognize, for example, a deviation of areference point of the host vehicle M from a center of the lane and anangle formed between a traveling direction of the host vehicle M and aline connecting the center of the lane as a relative position andposture of the host vehicle M with respect to the traveling lane.Instead, the recognizer 110 may recognize, for example, a position ofthe reference point of the host vehicle M with respect to any one ofside edge portions (the demarcation line or the road boundary) of thetraveling lane as the relative position of the host vehicle M withrespect to the traveling lane. The recognition of the traveling lane orthe recognition of the position or posture of the host vehicle M withrespect to the traveling lane by the recognizer 110 may be executed by aspecifier 153, which will be described below.

The recognizer 110, for example, realizes in parallel a function usingartificial intelligence (AI) and a function using a model given inadvance. For example, a function of “recognizing an intersection” may berealized by recognition of the intersection using deep learning or thelike and recognition based on previously given conditions (there is asignal which can be subjected to pattern matching, a road sign, or thelike) being executed in parallel and scored for comprehensiveevaluation.

The recognizer 110 analyzes the image captured by the driver monitorcamera 70 and recognizes a line-of-sight direction of the occupant(particularly, the driver) of the host vehicle M on the basis of ananalysis result.

The determiner 120 determines a traveling state (traveling mode) of thehost vehicle M to be executed by the driving controller 130 on the basisof a recognition result of the recognizer 110. Examples of the travelingmodes of the host vehicle M include at least a road traveling mode (anexample of a first mode) in which the host vehicle M travels in adirection in which the road extends (a direction in which the hostvehicle M can travel), an intersection mode (an example of a secondmode) in which the host vehicle M travels at an intersection, a mergingmode (an example of a third mode) in which the host vehicle M travels ina merging section, a narrow road passage mode (an example of a fourthmode) in which the host vehicle M passes a road whose width is smallerthan a predetermined value, and an emergency avoidance mode (an exampleof a fifth mode) for avoiding contact between the host vehicle M and anobject. The road traveling mode is an example of a “normal mode.” Themerging mode may include a mode (lane change mode) in which the hostvehicle M performs lane change from a traveling lane to an adjacentlane. The determiner 120 determines the traveling mode of the hostvehicle M on the basis of a positional relationship between the hostvehicle M and an object around the host vehicle M recognized by therecognizer 110 or road information around the host vehicle M obtainedfrom the map information 54. The determiner 120 may determine a displaymode associated with the traveling mode of the host vehicle M. Thedeterminer may determine whether or not the host vehicle M comes intocontact with a nearby object on the basis of a relative distance orrelative speed between the host vehicle M and the nearby object, amovement direction, or the like, or determine whether a degree ofapproach is equal to or greater than a threshold value. A function ofthe determiner 120 will be described in detail below.

The driving controller 130 controls a driving operation with respect tothe host vehicle M or the traveling of the host vehicle M by theoccupant of the host vehicle M. For example, the driving controller 130executes driving assistance corresponding to a traveling state of thehost vehicle M determined by the determiner 120. Examples of the drivingassistance include control for assigning a predetermined torque reactionforce to the steering wheel of the driving operator 80 in order to causethe occupant to execute a steering operation in a predetermineddirection, control for avoiding contact between the host vehicle M andan object, and control for changing the lane of the host vehicle M fromthe traveling lane to an adjacent lane. The driving controller 130causes the HMI controller 140 to output information according to thetraveling state of the host vehicle M from the HMI 30 so that theoccupant (especially, the driver) of the host vehicle M can driveappropriately according to the traveling state of the host vehicle M.

The HMI controller 140 notifies the occupant of predeterminedinformation using the HMI 30 or acquires content of an operation of theoccupant via the HMI 30. Examples of the predetermined informationinclude information related to traveling of the host vehicle M, such asinformation on a state of the host vehicle M or information on drivingcontrol. Examples of the information on the state of the host vehicle Minclude a speed of the host vehicle M, an engine speed, and a shiftposition. Examples of the information on the driving control include aninquiry as to whether or not to perform lane change, whether or not toexecute each of the traveling modes (the first to fifth modes) describedabove, information on changing the driving mode, and information imposedon the occupant required for switching between driving modes (taskrequest information for the occupant). The predetermined information mayinclude information irrelevant to travel control of the host vehicle M,such as TV programs, and content (for example, movies) stored in astorage medium such as a DVD. Examples of the predetermined informationmay include a current position or destination of the host vehicle M,information on a remaining amount of fuel, information indicatingwhether or not the traveling lane of the host vehicle M can bespecified, a remaining distance until the traveling mode is switched, anumber-of-lanes increase or decrease direction, the increased ordecreased number of lanes, and the number of lanes traveling in parallelto the traveling lane (the number of parallel traveling lanes).

For example, the HMI controller 140 may generate an image including thepredetermined information described above and cause the generated imageto be displayed on the display device 32 of the HMI 30, and may generatea sound indicating the predetermined information and cause the generatedsound to be output from a speaker of the HMI 30. For example, the HMIcontroller 140 causes an image in a display mode corresponding to anyone of the plurality of modes to be displayed on the display device 32on the basis of the surroundings situation recognized by the recognizer110. For example, the HMI controller 140 selects and generates an imageto be displayed among at least an image imitating the host vehicle M, animage showing a position of an object around the host vehicle M, animage showing a direction in which the host vehicle M should travel, atarget position that is a lane change destination of the host vehicle M,an image showing a direction that the occupant should watch, an imagefor prompting acceleration or deceleration, and an image for promptingthe occupant to perform a steering operation on the basis of the displaymode transitioning on the basis of the surroundings situation recognizedby the recognizer, and causes the generated image to be displayed on thedisplay device 32. The direction in which the host vehicle M shouldtravel or the direction that the occupant of the host vehicle M shouldwatch may be acquired on the basis of, for example, a position of theobject with respect to the host vehicle M recognized by a predeterminedrecognition model, or may be acquired by inputting, for example, acurrent surroundings situation of the host vehicle M to a learned modelthat has been learned from data such as driving content or aline-of-sight direction of a skilled driver according to a surroundingssituation included in a past driving history or the like. The HMIcontroller 140 may output information received by the HMI 30 to thecommunication device 20, the navigation device 50, the drivingcontroller 130, and the like.

The travel driving force output device 200 outputs a travel drivingforce (torque) for traveling of the vehicle to driving wheels. Thetravel driving force output device 200 includes, for example, acombination of an internal combustion engine, an electric motor, atransmission, and the like, and an electronic control unit (ECU) thatcontrols these. The ECU controls the above configuration according toinformation input from the driving assistance device 100 or informationinput from the driving operator 80.

The brake device 210 includes, for example, a brake caliper, a cylinderthat transfers hydraulic pressure to the brake caliper, an electricmotor that generates hydraulic pressure in the cylinder, and a brakeECU. The brake ECU controls the electric motor according to theinformation input from the driving assistance device 100 or theinformation input from the driving operator 80 so that a brake torqueaccording to a braking operation is output to each wheel. The brakedevice 210 may include, as a backup, a mechanism that transfershydraulic pressure generated by operating the brake pedal included inthe driving operator 80 to the cylinder via a master cylinder. The brakedevice 210 is not limited to the configuration described above, and maybe an electronically controlled hydraulic brake device that controls anactuator according to information input from the driving assistancedevice 100 and transmits the hydraulic pressure of the master cylinderto the cylinder.

The steering device 220 includes, for example, a steering ECU and anelectric motor. The electric motor, for example, changes orientations ofsteerable wheels by causing a force to act on a rack and pinionmechanism. The steering ECU drives the electric motor according to theinformation input from the driving assistance device 100 or theinformation input from the driving operator 80 to change theorientations of the steerable wheels. The steering device 220 may assigna torque reaction force so that the steering wheel is turned in apredetermined direction (so that the steering wheel is not turned in thepredetermined direction) according to a driving operation of theoccupant under the control of the driving assistance device 100.

[Determiner]

Next, a function of the determiner 120 will be described in detail. FIG.2 is a diagram for describing the function of the determiner 120. In theexample of FIG. 2 , five modes (the normal mode (the first mode), theintersection mode (the second mode), the merging mode (the third mode),the narrow road passage mode (the fourth mode) and the emergencyavoidance mode (the fifth mode)) indicating the traveling state of thehost vehicle M described above are shown. For example, the determiner120 determines that transition to a mode corresponding to apredetermined transition condition is performed when the predeterminedtransition condition is satisfied with reference to the normal mode(road traveling mode), and determines that transition to the normal modehas been performed when a predetermined return condition is satisfied inthe mode after the transition. When a determination is made that thetraveling state has transitioned to one mode, the driving controller 130causes the HMI controller 140 to generate an image associated with adisplay mode based on a determination result, and causes the generatedimage to be displayed on the display device 32. Hereinafter, thetransition condition and the return conditions to each mode shown inFIG. 2 will be described. Although a transition condition from thenormal mode to another mode and a return condition from the other modeto the normal mode are set in the example of FIG. 2 , a transitioncondition or a return condition between the respective modes may be setwithout passing through the normal mode.

<Normal Mode (Road Traveling Mode)>

The normal mode is a traveling mode in a case in which a determinationis made that a shape of a road (lane) on which the host vehicle M istraveling is not a predetermined road shape such as an intersection, amerging lane, or a narrow road, and a likelihood of contact between thehost vehicle M and another vehicle is low. The normal mode may be atraveling mode when a condition for transition to another mode is notsatisfied or when a condition for returning to another mode issatisfied. The determiner 120, for example, may acquire a shape of theroad on which the host vehicle M travels by referring to the mapinformation (the map information 54) on the basis of the positioninformation of the host vehicle M, and may acquire the shape on thebasis of a detection result of the external sensor ES.

When the normal mode is executed, the HMI controller 140 causes theimage imitating the host vehicle M to be displayed, causes the imageshowing the direction that the occupant should watch to be displayedwhen there is an object whose degree of approach to the host vehicle M(an index value indicating a likelihood of contact) is equal to orgreater than a threshold value (likelihood of contact is low but arelative distance is smaller than a predetermined distance), and causesan image showing a relative position of an object with respect to thehost vehicle M to be displayed when there is an object whose degree ofapproach to the host vehicle M is smaller than a threshold value (thelikelihood of the contact is low but the relative distance is equal toor larger than the predetermined distance).

For example, when another vehicle m1 is present in front of the hostvehicle M, an image showing a position of the other vehicle m1 withrespect to the position of the host vehicle M is displayed on thedisplay device 32, or a route for not coming into contact with the othervehicle m1 may be generated and displayed on the display device 32, asshown on the road RD1 in FIG. 2 . When there is an object (trafficparticipant) OB1 such as a bicycle overtaking the other vehicle m1, thedriving controller 130 may cause an image showing a position ordirection of the other vehicle m1 or the object OB1 with respect to theposition of the host vehicle M to be displayed on the display device 32or cause an image showing a route for overtaking the other vehicle m1 orthe object OB1 without coming into contact with the other vehicle m1 orthe object OB1 to be displayed on the display device 32, as shown on theroad RD2 in FIG. 2 . The driving controller 130 may control one or bothof steering and speed of the host vehicle M to perform driving controlso that the host vehicle M does not come into contact with the object.

<Intersection Mode>

A condition (transition condition A) for transition from the normal modeto the intersection mode is, for example, that there is an intersectionwithin a predetermined distance from the host vehicle M in the travelingdirection of the host vehicle M by referring to the map information (themap information 54) on the basis of the position of the host vehicle M.The transition condition A may be a case in which the blinker switch ofthe HMI 30 has been actuated according to an operation of the occupantand either a left blinker or a right blinker is turned on (is blinking),in addition to the above condition. The determiner 120 may determinewhether the host vehicle M is traveling on an intersection on the basisof the detection result of the external sensor ES, instead of (or inaddition) referring to the map information. When the determiner 120determines that transition to the intersection mode has been performed,the driving controller 130 causes the traveling mode of the host vehicleM to transition to the intersection mode.

When the transition to the intersection mode has occurred, the HMIcontroller 140 generates the image imitating the host vehicle M,generates an image showing a relative position of an object with respectto the host vehicle M when there is an object likely to come intocontact with the host vehicle M at the intersection, generates the imageshowing the direction that the occupant should watch when the object islocated in the traveling direction of the host vehicle M, and causes thegenerated image to be displayed on the display device 32.

For example, the HMI controller 140 causes an image showing the positionof the other object (traffic participant) OB1 such as a bicycle passingthrough the intersection when the host vehicle M turns right to bedisplayed on the display device 32, as shown on a road RD3 in FIG. 2 orcauses an image showing a position of another object (trafficparticipant) OB2 such as a bicycle passing near the intersection whenthe host vehicle M turns left to be displayed on the display device 32,as shown on the road RD4.

A condition for returning from the intersection mode to the normal mode(return condition A) is, for example, that the position of the hostvehicle M is separated from the intersection by a predetermined distanceor more. Further, the return condition A may include a case in which thehost vehicle M is not traveling in a merging section or a narrow roadand a likelihood of contact with another object is not high. The drivingcontroller 130 may control one or both of the steering and the speed ofthe host vehicle M to perform driving control so that the host vehicle Mdoes not come into contact with an object.

<Merging Mode>

A condition for transition from the normal mode to the merging mode(transition condition B) is, for example, a case in which the hostvehicle M is traveling on a road that includes a plurality of lanes inwhich the host vehicle M can travel in the same direction, or a case inwhich it is recognized that a lane in which the host vehicle M iscurrently traveling merges with another lane, which is a case in whichthe blinker switch of the HMI 30 is actuated according to an operationof the occupant and either the left blinker or the right blinker isturned on (blinks). The transition condition B may include a case inwhich the road on which the host vehicle M is traveling is not near theintersection, in addition to the above condition. Further, thedeterminer 120 may determine the mode to be the merging mode when thereis a merging section within the predetermined distance from the hostvehicle M in the traveling direction of the host vehicle M by referringto the map information on the basis of the position of the host vehicleM, and determine the mode to be the lane change mode when there is nomerging section within the predetermined distance. Further, thedeterminer 120 may determine whether or not the host vehicle M travelsin the merging section on the basis of the detection result of theexternal sensor ES, instead of (or in addition to) referring to the mapinformation.

When transition to the merging mode has occurred, the HMI controller 140causes the image imitating the host vehicle M to be displayed, causesthe other vehicle traveling in a lane that is a lane change destinationto be displayed as an image with a first predetermined color, and causesan image showing a target position of the lane change destination to bedisplayed as an image with a second predetermined color. The HMIcontroller 140 causes the image with the first predetermined color andthe image with the second predetermined color to be displayed side byside on the side in which the host vehicle M performs lane change withrespect to the image imitating the host vehicle M. The HMI controller140 may cause an image for accelerating or decelerating the host vehicleM on the basis of a positional relationship between the host vehicle Mand the target position to be displayed.

For example, the HMI controller 140 causes an image for assistingdriving for changing lanes from a merging lane to a main lane orchanging lanes from the traveling lane to an adjacent lane so that thehost vehicle M does not come into contact with other vehicles m1 and m2present around the host vehicle M, as shown on roads RD5 and RD6 in FIG.2 to be displayed on the display device 32.

The driving controller 130 may control one or both of the steering andthe speed of the host vehicle M to perform driving control so that thehost vehicle M does not come into contact with an object. The drivingcontroller 130 may perform control to guide the occupant to perform asteering operation for causing the host vehicle M to change lanes. Inthis case, the driving controller 130 performs an operation for guidingthe occupant to perform steering for lane change by using a reactionforce to the steering wheel operated by the occupant.

The condition for returning from the merging mode to the normal mode(return condition B) is, for example, a case in which the blinker isturned off. Another condition may be that the position of the hostvehicle M is separated from the merging section by a predetermineddistance or more. Furthermore, the return condition B may include a casein which the host vehicle M is not traveling near an intersection or ona narrow road and is not likely to come into contact with anotherobject.

<Narrow road passage Mode>

A condition (transition condition C) for transition from the normal modeto the narrow road passage mode is, for example, a case in which a widthof a road of the host vehicle M is smaller than a predetermined value.The transition condition C is, for example, a case in which the hostvehicle M travels while avoiding an object present in front of the hostvehicle M or a case in which the host vehicle M passes an oncomingtraveling object, which is a case in which a distance to the object issmaller than a predetermined distance and a time to collision TTCbetween the host vehicle M and the object is equal to or greater than athreshold value as shown on roads RD7 and RD8 in FIG. 2 . Further, thetransition condition C may include, for example, a case in which aminimum width of an area recognized as an area in which the host vehicleM can travel is smaller than a predetermined value (for example, ahorizontal distance from a left edge of the road to a leftmost edge of avehicle waiting for a right turn is smaller than a predetermined value).The time to collision TTC is, for example, a value that is calculated bydividing a relative distance by a relative speed in a relationshipbetween the host vehicle M and the object. The threshold value may be,for example, a fixed value or may be a variable value that is setdepending on the speed of the host vehicle M, the speed of the object, aroad situation, and the like. The transition condition C may includethat the speed of the host vehicle M is lower than a predeterminedspeed.

For example, the determiner 120 determines that the transition to thenarrow road passage mode occurs when another vehicle m1 stopping infront of the host vehicle M while waiting for a right turn or a leftturn is recognized and there is no adjacent lane on the left side of alane in which the host vehicle M (or the other vehicle m1) travels, asshown on the road RD8 in FIG. 2 . The determiner 120 determines that thetransition to the narrow road passage mode occurs when the other vehiclem1 is stopping while waiting for a left turn and there is no adjacentlane on the right side of the lane in which the host vehicle M istraveling, or when the adjacent lane is present but is underconstruction, or when there is no space for the host vehicle M to enterdue to a line of vehicles.

When transition to the narrow road passage mode has occurred, the HMIcontroller 140 causes the image imitating the host vehicle M to bedisplayed, and causes a direction in which the host vehicle M shouldtravel to be displayed and the image for prompting the occupant toperform a steering operation to be displayed when there is an objectaround the host vehicle M. For example, the HMI controller 140 causes animage for assisting driving for passing along the narrow road so thatthe host vehicle does not come into contact with the other vehicles m1and m2 present around the host vehicle M in a scene indicated by theroad RD7 or the road RD8 to be displayed on the display device 32. Thedriving controller 130 may execute steering control or speed control foravoiding contact with the other vehicles m1 and m2. The drivingcontroller 130 may perform control to guide the occupant to perform asteering operation for avoiding contact between the host vehicle M andan object. In this case, the driving controller 130 performs anoperation for guiding the occupant to perform steering by using thereaction force to the steering wheel operated by the occupant.

A condition for returning from the narrow road passage mode to thenormal mode (return condition C) is, for example, that a width of theroad on which the host vehicle M travels is equal to or greater than apredetermined value. The return condition C may include a case in whichthe host vehicle M and the other vehicle m1 pass each other and then areseparated by a predetermined distance or more, or a case in which thespeed of the host vehicle M is equal to or higher than a predeterminedspeed. Further, the return condition C may include a case in which theminimum width of the area recognized as the area in which the hostvehicle M can travel is equal to or greater than the predetermined value(for example, a case in which, when the host vehicle M tries to pass bya vehicle waiting for a right turn and the narrow road passage mode isactivated, the vehicle waiting for a right turn completes the rightturn, and the area in which the host vehicle M can travel increases andthe minimum width becomes equal to or greater than the predeterminedvalue).

<Emergency Avoidance Mode>

A condition for transition from the normal mode to the emergencyavoidance mode (transition condition D) is, for example, a case in whichthe time to collision TTC between the host vehicle M and an objectpresent near the host vehicle M is smaller than the threshold value.When the driving controller 130 executes driving assistance such asforward collision warning (FCW) when the time to collision TTC betweenthe host vehicle M and the object present near the host vehicle M issmaller than a first threshold value, or executes driving assistancesuch as collision mitigation braking system (CMBS) when the time tocollision TTC is smaller than a second threshold value that is smallerthan the first threshold value, the transition condition D may includethat any driving assistance is executed.

When transition to the emergency avoidance mode has occurred, the HMIcontroller 140 causes the image imitating the host vehicle M to bedisplayed, causes an image showing a direction in which the host vehicleshould travel with respect to the image imitating the host vehicle M tobe displayed, and causes an area other than the direction in which thehost vehicle M should travel around the image imitating the host vehicleM to be displayed in a background color different from those displayedin the first to fourth modes.

The HMI controller 140 causes an image for assisting driving foravoiding contact between the objects OB1 and OB2 such as bicyclesentering in front of the host vehicle M and the host vehicle M as shownon the road RD9 in FIG. 2 to be displayed on the display device 32. Thedriving controller 130 may control one or both of the steering and thespeed of the host vehicle M to perform driving control so that the hostvehicle M does not come into contact with an object.

Further, the condition for returning from the emergency avoidance modeto the normal mode (return condition D) is, for example, a case in whichthe time to collision TTC between the host vehicle M and the object isequal to or greater than the threshold value. The return condition D mayinclude a case in which the host vehicle M is not traveling at theintersection, the merging section, the narrow road, or the like.

[Images Displayed in Correspondence to Respective Modes]

Next, an example of images displayed in the respective modes describedabove will be described.

<Normal Mode: First Display Mode>

FIG. 3 is a diagram for describing the first display mode in the normalmode. In FIG. 3 , an image IM10A generated by the HMI controller 140 anddisplayed on the display device 32 at the time of execution of thenormal mode, and a traveling state of the host vehicle M in whichcontent shown in the image IM10A is displayed are shown. Display contentor layout included in the image IM10A is not limited thereto. The sameapplies to the description of the subsequent drawings.

In the example of FIG. 3 , it is assumed that the host vehicle M istraveling on a road RD11 including lanes L1 and L2 and sidewalks SW1 andSW2 in an extension direction. An area VA1 indicates a line-of-sightdirection of the occupant (driver) of the host vehicle M recognized bythe recognizer 110.

When the normal mode is being executed, the HMI controller 140 generatesthe image IM10A including a first image IM11 imitating the host vehicleM, and second images (azimuth images) IM12-1 to IM12-16 obtained bydividing a surrounding (360 degrees) of the host vehicle M into 16pieces according to a predetermined reference (for example, equalintervals). The first image IM11 in the normal mode is the imageimitating the host vehicle M viewed from behind. Each of the secondimages IM12-1 to IM12-16 includes an area AR1 closest to the hostvehicle M, an area AR3 farthest from the host vehicle M, and an area AR2between the areas AR1 and AR3. The second images IM12-1 to IM12-16 aredisplayed in a predetermined background color (first background color).

The HMI controller 140 causes the second images IM12-1, IM12-2, andIM12-16 associated with a direction in which the host vehicle M cantravel without the likelihood of contact with an object on the basis ofthe surroundings situation recognized by the recognizer 110 to bedisplayed in color different from the other second images IM12-3 toIM12-15 (first highlight color; for example, blue) when the host vehicleM is in a traveling state as shown in FIG. 3 . This makes it easier forthe occupant (driver) to recognize a direction in which the host vehicleM is unlikely to come into contact with an object.

<Normal Mode: Second Display Mode>

FIG. 4 is a diagram for describing a second display mode in the normalmode. In the example of FIG. 4 , a traveling state differs from thetraveling state shown in FIG. 3 in that the object (for example, apedestrian) OB1 is present on the sidewalk SW1. The object OB1 isrecognized as a pedestrian by the recognizer 110, and the pedestrian isless likely to enter the lane L1 from the sidewalk SW1. A relativedistance between the host vehicle M and the object OB1 is equal to orgreater than a predetermined distance. Therefore, the object OB1 isdetermined to be an object whose degree of approach to the host vehicleM is smaller than the threshold value. In this case, the HMI controller140 generates an image IM10B in which the area AR2 corresponding to adistance to the host vehicle M among the areas AR1 to AR3 of the secondimage IM12-14 in a direction in which the object OB1 is present amongthe second images IM12-1 to IM12-16 is displayed in a color (secondhighlight color; green, for example) that can be distinguished from thatof the other second images and the other areas, on the basis of arelative position of the object OB1 as viewed from the host vehicle Mand the direction in which the object OB1 is present. In the example ofFIG. 4 , since a likelihood of the pedestrian OB1 entering the lane L1from the sidewalk SW1 or a likelihood of the pedestrian OB1 coming intocontact with the host vehicle M is low, an image showing the position ofthe object OB1 is displayed in a second highlight color that lesshighlights as compared to the first highlight color in association witha relative position of the object OB1 with respect to the host vehicleM.

<Normal Mode: Third Display Mode>

FIG. 5 is a diagram for describing a third display mode in the normalmode. In the example of FIG. 5 , a display mode in a scene in which theother vehicle m1 is present in front of the host vehicle M on the laneL1 on which the host vehicle M travels, which is a scene in which adistance between the host vehicle M and the other vehicle m1 is equal toor greater than a predetermined distance, unlike the traveling stateshown in FIG. 3 , is displayed. In the example of FIG. 5 , a scene inwhich another vehicle present in a direction of the second image IM12-16is recognized on the basis of a recognition result of the recognizer 110is shown.

In this case, the HMI controller 140 generates an image IM10C in whichthe second images IM12-1 and IM12-2 showing the direction in which thehost vehicle M is unlikely to come into contact with an object among thesecond images IM12-1 to IM12-16 are displayed in the first highlightcolor, and the two areas AR2 and AR3 among the areas AR1 to AR3 of thesecond image IM12-16 corresponding to the direction of the other vehiclem1 when a line-of-sight direction of the occupant is a direction of theother vehicle m1 are displayed in a color (third highlight color:orange, for example) different from those of the second images IM12-1 toIM12-15. Since the other vehicle m1 is present on the lane L1 in whichthe host vehicle M travels, there is a likelihood that the other vehiclem1 will come into contact with the host vehicle M. Therefore, the twoareas are displayed in the third highlight color, making it possibleeasier to call occupant's attention, unlike a case in which one area isdisplayed. In the third display mode, when the line-of-sight directionof the occupant is not the direction of the other vehicle m1, the HMIcontroller 140 causes the second image IM12 corresponding to thedirection of the other vehicle m1 to be displayed in a fourth highlightcolor (for example, red), as shown in a sixth display mode of the normalmode, which will be described below.

<Normal Mode: Fourth Display Mode>

FIG. 6 is a diagram for describing a fourth display mode in the normalmode. In an example of FIG. 6 , a scene in which an instruction forcausing the host vehicle M to travel while being tilted to the rightwith respect to a direction in which the road RD11 extends in order forthe host vehicle M to overtake the other vehicle m1 unlike the travelingstate shown in FIG. 5 is displayed is shown. In the example of FIG. 6 ,it is assumed that a line-of-sight direction VA1 of the occupant of thehost vehicle M is a direction in which the other vehicle m1 is notincluded. In the scene shown in FIG. 6 , it is assumed that a degree ofapproach between the host vehicle and the other vehicle m1 is equal toor greater than the threshold value. In this case, the HMI controller140 causes the second image IM12-2 corresponding to a course directionto which the host vehicle M should travel among the second images IM12-1to IM12-16 to be displayed in the first highlight color. The coursedirection in which the host vehicle M should travel includes a directionin which the host vehicle M can travel with a likelihood of the hostvehicle M coming contact with an object (contact risk) being smallerthan a threshold value. The HMI controller 140 generates an image IM10Din which the two areas AR2 and AR3 among the areas AR1 to AR3 of imagesshowing the direction that the occupant should watch, which are thesecond images IM12-16 and IM12-1 corresponding to the direction of theother vehicle m1 viewed from the host vehicle M, are displayed in thethird highlight color. The number of areas displayed in the thirdhighlight color increases as compared with the third display mode,making it possible to more clearly notify the occupant that the othervehicle m1 is approaching the host vehicle M, and of the direction thatthe occupant should watch. Since the scene shown in FIG. 6 is a sceneafter the occupant once recognizes the other vehicle m1, the “directionthat the occupant should watch” may be read as a “direction to which theoccupant should call attention”.

<Normal Mode: Fifth Display Mode>

FIG. 7 is a diagram for describing a fifth display mode in the normalmode. In the example of FIG. 7 , a scene in which the host vehicle Movertakes the other vehicle m1 and then indicates a direction formovement to a center of the lane L1 unlike the traveling state shown inFIG. 6 is shown. In the example of FIG. 7 , it is assumed that theline-of-sight direction VA1 of the occupant of the host vehicle M is adirection in which the other vehicle m1 is not included.

In this case, the HMI controller 140 generates an image IM10E in whichthe second image IM12-16 corresponding to the direction in which thehost vehicle M should travel among the second images IM12-1 to IM12-16is displayed in the first highlight color, and one area AR2 setaccording to the distance between the host vehicle M and the othervehicle m1 among the areas AR1 to AR3 of the second image IM12-11corresponding to the direction in which the other vehicle m1 is presentas viewed from the host vehicle M is displayed in a third highlightcolor. In the fifth display mode, since the host vehicle M has alreadyovertaken the other vehicle m1, a likelihood of contact between the hostvehicle M and the other vehicle m1 is low (the degree of approach issmaller than the threshold value). Therefore, the HMI controller 140causes only one area AR2 corresponding to a position of the othervehicle m1 among the areas AR1 to AR3 of the second image IM12-11 to bedisplayed in the third highlight color. This makes it possible for theoccupant to easily ascertain that the likelihood of the contact betweenthe host vehicle M and the other vehicle m1 is low, but the othervehicle m1 is present near the host vehicle M.

<Normal Mode: Sixth Display Mode>

FIG. 8 is a diagram for describing the sixth display mode in the normalmode. In the example of FIG. 8 , a scene in which the other objects OB1and OB2 are present nearby in the traveling direction (the line-of-sightdirection VA1) of the host vehicle M. The other objects OB1 and OB2 are,for example, traffic participants such as pedestrians or bicycles. Theother objects OB1 and OB2 move in the lane L1 in which the host vehicleM travels and are present within the predetermined distance from thehost vehicle M. Therefore, the driving controller 130 determines thatthe degree of approach between the other objects OB1 and OB2 and thehost vehicle M is equal to or greater than the threshold value.

In this case, the HMI controller 140 generates an image IM10F in whichthe areas AR2 and AR3 displayed in two areas of the host vehicle M andthe other objects OB1 and OB2 among the areas AR1 to AR3 in the imagesshowing a direction that the occupant should watch among the secondimages IM12-1 to IM12-16, which are the second images IM12-1, IM12-2,and IM12-16, are displayed in a color (the fourth highlight color: red,for example) different from those of the other second images. The fourthhighlight color is a color that further highlights as compared to thefirst to third highlight colors. This makes it possible to notify theoccupant of directions in which there are the objects OB1 and OB2 thatare highly likely to come into contact with the host vehicle M andshould be watched by the occupant.

The driving controller 130 may execute speed control such as causing thehost vehicle M to decelerate or stop, in addition to causing the image10F described above to be displayed, in the scene shown in FIG. 8 . Inthis case, the driving controller 130 may release the deceleration orstop control when a distance between the host vehicle M and the objectsOB1 and OB2 is equal to or greater than a predetermined distance, orwhen the objects OB1 and OB2 have moved to positions other than the laneL1.

<Intersection Mode: First Display Mode>

FIG. 9 is a diagram for describing a first display mode in theintersection mode. In the example of FIG. 9 , a scene in which the hostvehicle M travels on a road RD12 near an intersection at which lanes L1and L2 and lanes L3 and L4 intersect, which is a scene in which the hostvehicle M turns right from the lane L1 to the lane L3 is shown. In theexample of FIG. 9 , it is assumed that an object OB1 present near theintersection in the line-of-sight direction VA of the occupant is notincluded, and the determiner 120 determines that the host vehicle M andthe object OB1 are likely to come into contact with each other on thebasis of relative distances, relative speeds, and moving directionsthereof.

When the host vehicle M travels near the intersection in theintersection mode, the HMI controller 140 generates an image IM10G inwhich the second image IM12 corresponding to the line-of-sight directionVA1 is not highlighted and the second image IM12-4 corresponding to aposition in which the object OB1 is present with respect to the hostvehicle M is displayed in the fourth highlight color so that the objectOB1 present around the host vehicle M is particularly easily recognized.In the example of FIG. 9 , only the area AR2 of the second image IM12-4is displayed in the fourth highlight color on the basis of the relativedistance between the host vehicle M and the object OB1, but the area AR1or the area AR3 may be displayed in the fourth highlight color accordingto the relative distance. Since this makes it possible for the occupantto be notified of the presence of the object nearby in the travelingdirection, it is possible to cause the line of sight of the occupant tobe guided in a direction in which the occupant should watch, and tocause the occupant to easily recognize the object OBE

<Intersection Mode: Second Display Mode>

FIG. 10 is a diagram for describing a second display mode in theintersection mode. In the example of FIG. 10 , the second display modeis different from the first display mode shown in FIG. 9 in that theobject OB1 is included in the line-of-sight direction of the occupant.In this case, the HMI controller 140 generates an image IM10H fordisplaying the second image IM12-3 corresponding to the position inwhich the object is present in the third highlight color having a lowerdegree of highlighting than the fourth highlight color.

In the example of FIG. 10 , only the area AR2 of the second image IM12-3according to the position of the object OB1 as viewed from the hostvehicle M is displayed in the third highlight color on the basis of arelative distance between the host vehicle M and the object OB1. The HMIcontroller 140 may display the area AR1 and the area AR3 of the secondimage IM12-3 in the third highlight color when the relative distance issmaller than a predetermined distance.

<Merging Mode: First Display Mode>

FIG. 11 is a diagram for describing a first display mode in the mergingmode. In the example of FIG. 11 , a scene in which the host vehicle Mperforms lane change from a lane L1 to a lane L2 on a road RD13, whichincludes the lanes L1 and L2 in which the host vehicle M can travel inthe traveling direction, is shown. In the example of FIG. 11 , it isassumed that other vehicles m1 and m2 are traveling on the lane L2,which is a lane change destination of the host vehicle M.

In this case, the driving controller 130 derives a relative speed andrelative position between the host vehicle M and each of the othervehicles m1 and m2, and sets a target position that is the lane changedestination of the host vehicle M on the basis of the derived relativespeed and relative position. The driving controller 130 causes the HMIcontroller 140 to generate an image for performing driving for movementto the target position without the host vehicle M contacting the othervehicles m1 and m2.

In the first display mode of the merging mode, the HMI controller 140generates an image IM20A including a third image IM21 imitating the hostvehicle M, fourth images (azimuth images) IM22-1 to IM22-12 obtained bydividing surroundings (360 degrees) of the host vehicle M into 12 piecesaccording to a predetermined reference (for example, equal intervals),fifth images IM23-1 and IM23-2 showing positions of other vehicles M,and a sixth image IM24 showing a target position TA1. The third imageIM12 is the image imitating the host vehicle M viewed from above. Eachof the fourth images IM22-1 to IM22-12 includes an area AR1 closest tothe host vehicle M, an area AR3 farthest from the host vehicle M, and anarea AR2 between the areas AR1 and AR3.

The fifth image IM23 and the sixth image IM24 only need to be displayedin an identifiable display mode. The fifth image IM23 is an example ofthe “image with first predetermined color”, and the sixth image IM24 isan example of the “image with second predetermined color”. The fifthimage IM23 and the sixth image IM24 are displayed side by side atpositions on the lane change side of the host vehicle M with respect tothe third image imitating the host vehicle M, and at least a partthereof may be displayed to be superimposed on the fourth image IM22. Inthe merging mode, the fourth image IM22 includes information indicatingthe acceleration or deceleration of the host vehicle M.

In the merging mode, the fifth images IM23-1 and IM23-2 showing thepositions of the other vehicles M and the sixth image IM24 indicatingthe target position TA1 are displayed with reference to the position ofthe third image IM21 imitating the host vehicle M, making it possiblefor the occupant to easily recognize the current target position TA1with respect to the host vehicle M.

In the example of FIG. 11 , the position of the host vehicle M ispresent behind the target position TA1 by a predetermined distance ormore. Therefore, the HMI controller 140 generates an image for promptingthe occupant to accelerate the host vehicle M. Specifically, the fourthimage IM22-1 showing the front of the host vehicle M among the fourthimages IM22-1 to IM22-12 is displayed in a color different from those ofthe other fourth images IM22-2 to IM22-12 to prompt the acceleration ofthe host vehicle M. In the example of FIG. 11 , since the host vehicle Mand the target position TA1 are separated by the predetermined distanceor more, all the areas AR1 to AR3 of the fourth image IM22-1 aredisplayed in different colors to prompt a large acceleration. In theexample of FIG. 11 , the area AR1 is displayed in the darkest color, andthe area AR2 and the area AR3 are displayed in gradually lighter colors,but the display mode is not limited thereto.

<Merging Mode: Second Display Mode>

FIG. 12 is a diagram for describing a second display mode in the mergingmode. In the example of FIG. 12 , the position of the host vehicle M iscloser to the target position than in the example of FIG. 11 . In thiscase, the target position TA1 is still in front of the host vehicle M.Therefore, the HMI controller 140 generates an image IM20B in which onlythe area AR1 included in the fourth image IM22-1 showing the front ofthe host vehicle M is displayed in a color different from that of theother fourth images IM22-2 to IM22-12. This makes it possible to easilyrecognize that the host vehicle M is approaching the target position TA1by using only a small amount of acceleration left, unlike the firstdisplay mode of FIG. 11 .

<Merging Mode: Third Display Mode>

FIG. 13 is a diagram for describing a third display mode in the mergingmode. In the example of FIG. 13 , the target position is present besidethe host vehicle M, unlike the example of FIG. 12 . In this case, sinceaddition acceleration of the host vehicle M is not required, the HMIcontroller 140 stops displaying the fourth image IM22-1 differently fromthe other images IM22-2 to IM22-12. This makes it possible for theoccupant to easily ascertain that the host vehicle M is positionedbeside the target position TA. Therefore, the occupant can position thehost vehicle M at the target position by steering the host vehicle M andchanging the lane of the host vehicle M from the lane L1 to the lane L2.When the display mode is brought to the third display mode shown in FIG.13 (that is, when the position of the host vehicle M is positionedbeside the target position TA1), the HMI controller 140 may cause theHMI 30 to output information (for example, an image or sound) forinstructing the occupant to change lanes from the lane L1 to the laneL2. The driving controller 130 may add a torque reaction force forturning the steering wheel operated by the occupant of the host vehicleM to the right (does not turn the steering wheel to the left).

<Merging Mode: Fourth Display Mode>

FIG. 14 is a diagram for describing a fourth display mode in the mergingmode. In the example of FIG. 14 , a case in which the target positionTA1 with respect to the host vehicle M is behind the host vehicle M by apredetermined distance or more is shown, unlike the example of FIG. 11 .In this case, the HMI controller 140 generates an image IM20D that is animage for prompting the occupant to decelerate the host vehicle M, andin which the areas AR1 to AR3 of the fourth image IM22-7 positionedbehind the third image IM21 imitating the host vehicle M among thefourth images IM22-1 to IM22-12 are displayed in colors different fromthose of the other fourth images. The HMI controller 140 causes thefifth images IM23-1 and IM23-2 and the sixth image IM24 to be displayedside by side at positions on the lane change side of the third imageIM21 imitating the host vehicle M is displayed, which are positionscorresponding to a relative distance from the host vehicle M, inassociation with relative positions of the other vehicles m1 and m2 andthe target position TA1 with respect to the host vehicle M.

This makes it possible for the occupant to easily ascertain from theimage IM20D that the host vehicle M needs to decelerate in order tochange lanes. The HMI controller 140 may reduce the number of areas tobe highlighted in a predetermined color among the areas AR1 to AR3 asthe relative distance between the host vehicle M and target position TA1decreases. This makes it easier for the occupant to ascertain how muchthe vehicle is to be decelerated from a current speed (decelerationlevel).

<Narrow Road Passage Mode: First Display Mode>

FIG. 15 is a diagram for describing a first display mode in the narrowroad passage mode. In an example of FIG. 15 , the host vehicle Mtraveling on a road RD14 determined to have a road width W1 smaller thana predetermined width and to be a narrow road, and another vehicle m1traveling on the road RD14 while facing the host vehicle M are shown. Inthe example of FIG. 15 , it is assumed that walls (examples ofobstacles) WL1 and WL2 through which the vehicle cannot pass are presentoutside both edges of the road RD14.

In the first display mode of the narrow road passage mode, the HMIcontroller 140 generates an image IM30A including a seventh image IM31imitating the host vehicle M and eighth images (azimuth images) IM32-1to IM32-12 obtained by dividing the surroundings (360 degrees) of thehost vehicle M into 12 pieces according to a predetermined reference(for example, equal intervals), and an image IM40A including images IM41and IM42 imitating a steering wheel. The image IM40A is an example ofthe image for prompting the occupant to perform a steering operation.The seventh image IM31 is an image of the host vehicle M viewed fromabove similar to the third image IM31, but is an image larger than thethird image IM21. In the narrow road passage mode, a situation fartherfrom the host vehicle may not be regarded as an important situationsince the host vehicle travels at a low speed, and a likelihood ofcontact with an object is high since the host vehicle travels in anarrower area. Therefore, the image (the seventh image IM31) imitatingthe host vehicle M in the display mode in the narrow road passage modeis displayed to be larger than the image (the third image IM21)imitating the host vehicle M in the merging mode, making it possible tomore accurately ascertain a position of a nearby object or a directionin which the host vehicle M should travel, with reference to the hostvehicle M, and call occupant's attention to the necessity of fineoperations in a short-range area as compared to other modes.

Each of the eighth images IM32-1 to IM32-12 includes the three areas AR1to AR3 described above. The image IM40A includes a first steering imageIM41 that is displayed according to a current steering amount andsteering direction of the host vehicle M, and a second steering imageIM42 that is displayed according to a steering amount and steeringdirection according to a direction in which host vehicle M shouldtravel. The HMI controller 140 may cause the images IM30A and IM40A tobe displayed within a predetermined range, or may cause the images IM30Aand IM40A to be displayed on different display devices when the hostvehicle M includes a plurality of display devices 32.

In the example of FIG. 15 , a determination is made that the hostvehicle M is approaching the other vehicle m1 and is likely to come intocontact with the other vehicle m1. Therefore, the HMI controller 140cause all the areas AR1 to AR3 of the eighth image IM32-12 showing thedirection in which the host vehicle M should travel among the eighthimages IM32-1 to IM32-12 to be displayed in a color different from thoseof the eighth other images IM32-1 to IM32-12, so that the occupant movesthe host vehicle M to the left side.

The HMI controller 140 causes the second steering image IM42 to beturned to the left relative to the first steering image IM41 anddisplayed in the image IM40A to cause the occupant to turn the steeringwheel of the host vehicle M to the left. This makes it possible toprompt the occupant to turn the steering wheel of the host vehicle M tothe left. Therefore, the occupant can turn the steering wheel to theleft and easily ascertain an amount of turning (an amount of steering).In the narrow road passage mode, the driving controller 130 may performcontrol for guiding the occupant to perform a steering operation foravoiding contact between the host vehicle M and the other vehicle m1. Inthis case, the driving controller 130 performs an operation for guidingthe occupant to perform steering so that the host vehicle M moves in thedirection in which the host vehicle M should travel, by using thereaction force to the steering wheel operated by the occupant.

<Narrow Road Passage Mode: Second Display Mode>

FIG. 16 is a diagram for describing a second display mode in the narrowroad passage mode. In the example of FIG. 16 , a scene differs from thescene shown in FIG. 15 in that the host vehicle M moves to the leftaccording to a steering operation of the occupant. In this case, anactual steering amount and steering direction of the steering wheel ofthe host vehicle M match a steering amount and steering direction of thesteering wheel associated with the direction in which the host vehicle Mshould travel. Therefore, in the image IM40B generated by the HMIcontroller 140, only the first steering image IM41 is displayed becausethe first steering image IM41 is displayed to be superimposed on thesecond steering image IM42. In the image IM30B generated by the HMIcontroller 140, all the areas AR1 to AR3 of the eighth image IM32-1corresponding to the line-of-sight direction of the occupant (the frontside of the host vehicle M) are displayed in a different color fromthose of the other eighth images IM32-2 to IM32-12.

This makes it easier for the occupant to ascertain that the host vehicleM is traveling in a direction in which the host vehicle M does not comeinto contact with the other vehicle m1.

<Narrow Road Passage Mode: Third Display Mode>

FIG. 17 is a diagram for describing a third display mode in the narrowroad passage mode. A screen shown in FIG. 17 shows a scene in which thehost vehicle M has moved to the left and passes another vehicle m1, butis approaching a wall WL1 present outside an edge portion of the roadRD14 (a distance between the host vehicle M and the wall WL1 is smallerthan a predetermined distance). In this case, the HMI controller 140generates an image IM30C in which the area AR1 closer to the hostvehicle M among the areas of an image showing a direction that theoccupant should watch, which is the eighth image IM32-12 correspondingto a direction in which there is a wall viewed from the host vehicle Mis displayed in a different color, and all the areas AR1 to AR3 of theeighth image IM32-2 showing the direction in which the host vehicle Mshould travel (specifically, a direction for avoiding contact betweenthe host vehicle M and the wall WL1) are displayed in a color differentfrom those of the other areas. The HMI controller 140 generates an imageIM40C including the first steering image IM41 with a position of theentire steering wheel as a reference, and a second steering image 42 forturning the steering wheel in the direction in which the host vehicle Mshould travel.

Thus, the images IM30C and IM40C are displayed on the display device 32,making it possible to notify that the host vehicle M is approaching thewall WL1, and prompt the occupant to perform a steering operation withrespect to the host vehicle M.

<Emergency Avoidance Mode: First Display Mode>

FIG. 18 is a diagram for describing a first display mode in theemergency avoidance mode. FIG. 18 shows the host vehicle M traveling ina lane L1 of a road RD15 including lanes L1 and L2 and sidewalks SW1 andSW2, and an object (bicycle) OB1 entering the lane L1 from the sidewalkSW1. In an example of FIG. 18 , it is assumed that the transitioncondition D for transition to the emergency avoidance mode is satisfied.

In the first display mode of the emergency avoidance mode, the HMIcontroller 140 generates an image IM50A including a ninth image IM51imitating the host vehicle M, and tenth images (azimuth images) IM52-1to IM52-16 obtained by dividing a circumference (360 degrees) of thehost vehicle M into 16 pieces according to a predetermined reference(for example, equal intervals). The ninth image IM51 in the normal modeis the image imitating the host vehicle M viewed from behind, like thefirst image IM11. Each of the second images IM12-1 to IM12-16 includesthe three areas AR1 to AR3, as described above.

In the case of the emergency avoidance mode, the HMI controller 140causes all the areas AR1 to AR3 of the ninth image IM52-4 showing adirection for avoiding the contact between the host vehicle M and theobject OB1 to be displayed in a different color from those of the otherninth images IM52-1 to IM52-3 and IM52-5 to IM52-16. The HMI controller140 causes the ninth images IM52-1 to IM52-3 and IM52-5 to IM52-16 to bedisplayed in a second background color different from the firstbackground color described above. The second background color is a colorthat further highlights as compared to the first background color. Asshown in FIG. 18 , the entire circumference of the host vehicle M isdisplayed in the second background color, making it possible for theoccupant to easily ascertain that there is an object highly likely tocome into contact with the host vehicle M.

In the emergency avoidance mode, the HMI controller 140 does not causethe image IM40 regarding the steering amount and steering direction ofthe steering wheel to be displayed, as in the narrow road passage mode.This makes it possible to suppress occupant's attention to driving beingdistracted, by a plurality of images being displayed.

[Processing Flow]

FIG. 19 is a flowchart showing an example of a flow of processing thatis executed by the driving assistance device 100 of the embodiment. Inthe process of FIG. 19 , image display processing according to the modeamong the various processes executed by the driving assistance device100 will be mainly described. The process of FIG. 19 may be repeatedlyexecuted at a predetermined cycle or at a predetermined timing. It isassumed that a first mode in the processing of FIG. 19 is the normalmode (road traveling mode).

In the example of FIG. 10 , the recognizer 110 recognizes thesurroundings situation of the host vehicle M (step S100). Next, thedeterminer 120 performs processing for determining a mode in thetraveling state of the host vehicle M on the basis of a recognitionresult of the recognizer 110 (step S102). For example, the determiner120 determines whether or not any of transition conditions (for example,transition conditions A to D) associated with a plurality of presetmodes is satisfied (step S104). When any transition condition issatisfied, the driving controller transitions to the mode associatedwith the transition condition (step S106). Next, the HMI controller 140generates an image associated with the mode after the transition (stepS108). The generated image is displayed on the display device 32 and theoccupant is notified of the image (step S110).

In the processing of step S104, when a determination is made that anytransition condition is not satisfied, the HMI controller 140 generatesan image associated with the normal mode (step S112), and causes thegenerated image to be displayed on the display device 32 and theoccupant to be notified of the generated image (step S114). Thus, theprocessing of the present flowchart ends.

In the processing of FIG. 19 , when a condition of transition from thenormal mode is satisfied, transition to a mode corresponding to thetransition condition occurs and then, when a predetermined returncondition is satisfied, return to the normal mode occurs. In this case,the HMI controller 140 generates an image in a display modecorresponding to the normal mode and causes the image to be displayed onthe display device 32, thereby notifying the occupant.

Modification Example

In the display of each image described above, the HMI controller 140 maycause the image to be highlighted through adjustment of gradation,pattern, blinking, luminance, brightness, or the like instead of causingthe image to be highlighted according to color.

According to the embodiment described above, the driving assistancedevice 100 includes the recognizer 110 configured to recognize asurroundings situation of a host vehicle; and the HMI controller (anexample of a display controller) 140 configured to cause an image forassisting an occupant driving the host vehicle to be displayed in aplurality of preset display modes on the display device, wherein theplurality of display modes include display modes corresponding to atleast a first mode in which the host vehicle travels in a direction inwhich the road extends, a second mode in which the host vehicle travelsat an intersection, a third mode in which the host vehicle travels in amerging section, a fourth mode in which the vehicle travels on a narrowroad, and a fifth mode in which the host vehicle travels while avoidingcontact with objects, and the HMI controller 140 causes transition toany one of the plurality of display modes to be performed on the basisof the surroundings situation recognized by the recognizer 110, therebyperforming more appropriate driving assistance according to thesurroundings situation.

The embodiment described above can be expressed as follows.

A driving assistance device includes

-   -   a storage device that stores a program, and    -   a hardware processor,    -   wherein the hardware processor executes the program to:    -   recognize a surroundings situation of a host vehicle;    -   cause an image for assisting an occupant driving the vehicle to        be displayed in a plurality of preset display modes on a display        device, the plurality of display modes including display modes        corresponding to at least a first mode in which the vehicle        travels in a direction in which the road extends, a second mode        in which the vehicle travels at an intersection, a third mode in        which the vehicle travels in a merging section, a fourth mode in        which the vehicle travels on a narrow road, and a fifth mode in        which the vehicle travels while avoiding contact with objects;        and    -   cause transition to any one of the plurality of display modes to        be performed on the basis of the recognized surroundings        situation.

While forms for carrying out the present invention have been describedusing the embodiments, the present invention is not limited to theseembodiments at all, and various modifications and substitutions can bemade without departing from the gist of the present invention.

What is claimed is:
 1. A driving assistance device comprising: arecognizer configured to recognize a surroundings situation of avehicle; and a display controller configured to cause an image forassisting an occupant driving the vehicle to be displayed in a pluralityof preset display modes on a display device, wherein the plurality ofdisplay modes include display modes corresponding to at least a firstmode in which the vehicle travels in a direction in which the roadextends, a second mode in which the vehicle travels at an intersection,a third mode in which the vehicle travels in a merging section, a fourthmode in which the vehicle travels on a narrow road, and a fifth mode inwhich the vehicle travels while avoiding contact with objects, and thedisplay controller causes transition to any one of the plurality ofdisplay modes to be performed on the basis of the surroundings situationrecognized by the recognizer.
 2. The driving assistance device accordingto claim 1, wherein the display controller selects an image to bedisplayed, from among at least an image imitating the vehicle, an imageshowing a position of an object around the vehicle, an image showing adirection in which the vehicle should travel, a target position servingas a lane change destination of the vehicle, an image for promptingacceleration or deceleration, and an image for prompting the occupant toperform a steering operation on the basis of a display modetransitioning on the basis of the surroundings situation recognized bythe recognizer, and causes the image to be displayed.
 3. The drivingassistance device according to claim 2, wherein the display controllercauses an image imitating the vehicle in the third mode and the fourthmode to be displayed as an image when the vehicle is viewed from above,and causes an image imitating the vehicle in the first mode, the secondmode, and the fifth mode to be displayed as an image when the vehicle isviewed from behind.
 4. The driving assistance device according to claim1, wherein the display controller causes an image imitating the vehicleto be displayed when the display controller causes an image to bedisplayed in a display mode corresponding to the first mode, and causesan image showing a direction to be watched by the occupant to bedisplayed when an object is present around the vehicle and the object isnot present in a line-of-sight direction of the occupant, and causes animage showing a direction of the object viewed from the vehicle to bedisplayed when the object is present in the line-of-sight direction ofthe occupant.
 5. The driving assistance device according to claim 1,wherein the display controller causes an image imitating the vehicle tobe displayed when the display controller causes an image to be displayedin a display mode corresponding to the second mode, causes an imageshowing a relative position of an object with respect to the vehicle tobe displayed when there is the object likely to come into contact withthe vehicle at the intersection, and causes an image showing a directionto be watched by the occupant to be displayed when the object is locatedin a traveling direction of the vehicle M.
 6. The driving assistancedevice according to claim 1, wherein the display controller causes animage imitating the vehicle to be displayed when the display controllercauses an image to be displayed in a display mode corresponding to thethird mode, causes another vehicle traveling in a lane serving as a lanechange destination to be displayed as an image with a firstpredetermined color and causes an image showing a target position of thelane change destination to be displayed as an image with a secondpredetermined color, and causes the image with the first predeterminedcolor and the image with the second predetermined color to be displayedon the side in which the vehicle performs lane change with respect tothe image imitating the vehicle
 7. The driving assistance deviceaccording to claim 6, wherein the display controller causes an image foraccelerating or decelerating the vehicle on the basis of a positionalrelationship between the vehicle and the target position to be displayedwhen the display controller causes the image to be displayed in thedisplay mode corresponding to the third mode.
 8. The driving assistancedevice according to claim 1, wherein the display controller causes animage imitating the vehicle to be displayed when the display controllercauses an image to be displayed in a display mode corresponding to thefourth mode, and causes a direction in which the vehicle should travelto be displayed and causes an image for prompting the occupant toperform a steering operation to be displayed when there is an objectaround the vehicle.
 9. The driving assistance device according to claim1, wherein the display controller causes an image imitating the vehicleto be displayed when the display controller causes an image to bedisplayed in a display mode corresponding to the fifth mode, and causesan image showing a direction in which the vehicle should travel withrespect to the image imitating the vehicle to be displayed, and causesan area other than the direction in which the vehicle should travelaround the image imitating the vehicle to be displayed in a backgroundcolor different from those displayed in the first to fourth modes.
 10. Adriving assistance method comprising: recognizing, by a computer, asurroundings situation of a vehicle; causing, by the computer, an imagefor assisting an occupant driving the vehicle to be displayed in aplurality of preset display modes on a display device, the plurality ofdisplay modes including display modes corresponding to at least a firstmode in which the vehicle travels in a direction in which the roadextends, a second mode in which the vehicle travels at an intersection,a third mode in which the vehicle travels in a merging section, a fourthmode in which the vehicle travels on a narrow road, and a fifth mode inwhich the vehicle travels while avoiding contact with objects; andcausing, by the computer, transition to any one of the plurality ofdisplay modes to be performed on the basis of the recognizedsurroundings situation.
 11. A computer-readable non-transitory storagemedium having a program stored therein, the program causing a computerto: recognize a surroundings situation of a vehicle; cause an image forassisting an occupant driving the vehicle to be displayed in a pluralityof preset display modes on a display device, the plurality of displaymodes including display modes corresponding to at least a first mode inwhich the vehicle travels in a direction in which the road extends, asecond mode in which the vehicle travels at an intersection, a thirdmode in which the vehicle travels in a merging section, a fourth mode inwhich the vehicle travels on a narrow road, and a fifth mode in whichthe vehicle travels while avoiding contact with objects; and causetransition to any one of the plurality of display modes to be performedon the basis of the recognized surroundings situation.